Skip to content

test: run test modules headless unless they need a workbench - #1622

Draft
joaodinissf wants to merge 1 commit into
masterfrom
headless-harness-step-2
Draft

joaodinissf wants to merge 1 commit into
masterfrom
headless-harness-step-2

Conversation

@joaodinissf

Copy link
Copy Markdown
Collaborator

Summary

Test modules now run under Tycho's default headless harness, so a module only starts a workbench if it asks for one. Based on #1599, which should merge first.

  • Seven modules need a workbench and opt in. They set tycho.surefire.useUIHarness=true and test.reactorConcurrencyLevel=1, so they take turns under -T: check.core.test, check.ui.test, test.ui.test, checkcfg.core.test, check.test.runtime.tests, sample.helloworld.ui.test, xtext.ui.test.
  • Seven modules run headless, with no lock: test, typesystem.test, xtext.export.test, check.runtime.core.test, xtext.test, xtext.generator.test, xtext.format.test.
  • A new test module runs headless by default and fails loudly if it needs a workbench: PlatformUI.getWorkbench() throws, and the Check injector can't be created without an IWorkbench. AGENTS.md explains how to opt in.

Performance

Local builds on macOS with Temurin 21. The master and #1599 figures come from earlier runs on older bases, and other work was running on the machine, so small differences are noise.

Build master #1599 before its lock #1599 with its lock This PR
Full goal list, -T 3C not measured 186–208 s 245–272 s 213–233 s
Tests only, -T 3C 105–109 s 77–81 s not measured 118 s (one run)
Tests only, serial 127 s 187–189 s same as before its lock 179 s, 206 s
  • Serial Surefire time drops from 172 s to 143 s, measured back to back on the same day.
  • Serial builds, which is what CI runs, stay about 50 s slower than master. That comes from one runtime per module in test: run each test bundle in its own runtime with JUnit discovery #1599; this PR recovers part of it.
  • Skipping the lock in the headless modules saves nothing measurable today: 213–233 s against 223–228 s when all 14 queue. The build's dependency order runs most of them before the workbench modules, which then set the end of the build. It stays because those modules don't need the lock, so future headless tests can run in parallel.

Validation

  • ./mvnw -U -T 3C clean verify checkstyle:check pmd:pmd pmd:cpd pmd:check pmd:cpd-check spotbugs:check -f ddk-parent/pom.xml --batch-mode --fail-at-end passes.
  • All 455 tests are present with the same statuses as before, including the 2 skipped, in every run.
  • At most 1 workbench JVM ran at a time, and headless JVMs ran alongside it for only about 11 s per build.
  • The headless modules have only run on macOS so far, so CI on this PR is their first Linux run.

Known limits

  • Two runs with headless modules outside the lock can't rule out a rare flake.
  • checkcfg.core.test hit a ConcurrentModificationException in validation once when tried headless. It's unexplained, and it hasn't recurred with that module on the UI harness.

🤖 Generated with Claude Code

@joaodinissf
joaodinissf added this pull request to stack #1623 November 6, 2026 10:43
@joaodinissf
joaodinissf force-pushed the headless-harness-step-2 branch from b66ed24 to 402b761 Compare November 15, 2026 19:07
Base automatically changed from test-discovery to master November 16, 2026 08:25
Test modules use Tycho's default headless harness. A module that needs a
workbench opts in by setting tycho.surefire.useUIHarness=true, and fails
loudly otherwise, so a new module does not start a workbench by accident.

Seven modules need one and opt in: check.core.test, check.ui.test,
test.ui.test, checkcfg.core.test, check.test.runtime.tests,
sample.helloworld.ui.test and xtext.ui.test. Only these take the Surefire
reactor lock (test.reactorConcurrencyLevel=1), since a workbench could take
focus from a SWTBot test in another one. The other seven have no workbench
and run in parallel under -T.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joaodinissf
joaodinissf force-pushed the headless-harness-step-2 branch from 402b761 to a28f0a9 Compare November 16, 2026 08:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant